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MEMORANDUM FOR: Acting Director 


SUBJECT : Consultants Report on the Estimative Process 


1, For your and the DCI's eyes onl 
am a bit disappointed in the report that 25X1 
ee oe have subiilnte don Ts Satins 
; suggested has been previously suggested 
on several occasions. For example: 
{1) Cutting down the number of coordinated papers. 
(2) Keeping a wide variety of views on the Board. 
{3} Reflecting diverse views. 
(4) Calling in consultants (which we do with Hyland and others). 


(7) Making more information about foreign intelligence 
organizations available. 


(9) Making validity studies in depth. 


On the other hand, I do think they have valid points in their technical items 


_ebout (5) indicating when earller eatimates are quoted, and (6) being bolder 


in estimating the impact of U.S. policy. Their point in (8) on signs that a 
mutation in an existing pattern is taking place I think is very important. 


wre F 2 B4 eater 
2. The annex on 'Preconceived Notions' I am afraid is a bit esdeaes 

as perhaps is the entire study, because these fellows are so closely 

associated with the Board of Estimates. 


ans ga 
eal Past, 
VEETS 1 


Bag Las 
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3, Having read this report 8 couple of times, I am disturbed that 
perhaps there is slippage in getting these done because we have been on 
top of Sherman Kent for months to bring in new blood on the Board, which 
Ne ge done withL____________ Pp to set-up a specific tour of 
duty so that there can be ease in rotation. On the other hand, I am still 
very concerned that the Board is "too much of the same, " and I would 
dearly like to see & couple of real hair-shirts from Missouri {to mix @ 
metaphor) to check some of the bureaucratic, governmental, acaderic 
views that lam sure consistently arise in Board discussions. 


Lyman B. Kirkpatrick 
Executive Director 


LBK:drm 
Distribution: 
Origina].- Addressee w/basic 
D ExDir 
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22 July 1963 


Dear Mr. McCone: 
I enclose the report of the committee of consultants which 
you appointed to examine the estimative process in the light of 
the Cuban experience and to make recommendations for improving 
it. 
25X1 


incere 


Contents: 


Introduction 

Recommendations 

Annex A, "Preconceived Notions" 
Annex B, Rejected Proposals 
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INTRODUCTION 


1, Onl? May, 1963, the Director of Central Intelligence asked 


25X1 


25X1 


to examine the estimative process in the light of the 
Cuban experience, and to make recommendations for improving it. 
25X1 | received a form of consultant's contract not compatible 
with his other committments and this mistake was not corrected in time 
to make it possible for him to serve on the committee, The other three 
members of the committee met for three days (8, 9, and 10 July 1963), 
in the office of ONE, exchanged draft proposals during the next week, 
and met again on 18 July to discuss the final form of their report. 

2. It is obvious that no thorough investigation of the estimative 
process could be made in this brief period. A number of estimates 
dealing with Cuba and the USSR were read, two development files were 
studied, and a few members (or former members) of ONE were inter- 
viewed, As consultants, the members of the committee have read, over 
the past few years, a number of estimates, both in final and in draft form, 

25X1 In addition, [was a member of the Board of National Estimates 
25X14 [Neve rthelesss, none of us feels that he 
is in any position to make emphatic and final judgments about existing 


estimative procedures, 


GROUP 1 
Excluded from automatic 
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3, Two of the committee believe that a study in depth of the 
estimative process by a small group of informed outsiders (not our - 
selves, we hasten to add) might yield valuable results, fCucha study 
would require access to all constituents of the intelligence community 
and knowledge of personnel as well as procedures, It would take 
several weeks, if not months, For these reasons the third member of 
the committee believes the project is not feasible, and he doubts that 
it would be useful. 

4, The discussion and suggestions which follow are only opinions 
offered by partially informed outsiders, Perhaps we are fairly typical 
consumers, and our report might be taken as representing the reactions 
of some consumers to estimates. But we should emphasize that we have 
not studied the problem in depth, and that therefore many of our sugges- 
tions may be superfluous, impractical, or just wrong-headed. We should 
also emphasize that our suggestions deal almat entirely with details of 
procedure, and that even the best procedure will not guarantee good 
estimates. The essential thing is to find and retain men who have wide 
knowledge, broad vision and good judgment, As long as the intelligence 


community has such men it will do a good job, whatever its procedures, 


25X1 
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The best way to remedy any deficiencies which occur in the 
estimative process is to have full and careful discussion of each prob- 
lem. In theory the present machinery should ensure such a discussion. 
In practice, we believe that there are some obstacles and we offer the 
following suggestions as means of reducing or removing these obstacles, 

1) Cut down the number of coordinated papers. 

Men who have too much to do in too short a time may not 

see all possibilities or sense the full impact of changed 

conditions, Pressure to meet deadlines forces limitation 

of discussion, Effort devoted to trivial papers means less 

time and energy for major problems, It would require more 

knowledge thak we possess to determine what papers should 

be eliminated, but we suspect that many ‘country papers" 

fall in this category. We also suspect that requests for 

coordinated papers are made, and accepted, too easily, 

But in any case, there is overloading and it has bad selfs 

(Note that we do not recommend any increase in 
personnel, The number of people now involved in the 
estimative process is about as large as can be used 
efficiently, More people would simply mean more spe- 
cialization and more meetings. What is needed is the 


me: 3 eee 
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overall view and time in which tog develop it and 
discuss it. This can be done only by a small group 
of carefully selected men), 

2) Make sure that there are always men with a 
wide variety of views and experience on the Board of 
National Estimates, The present distribution of member- 
ship on the Board seems well designed to achieve this end. 
As we understand it, it includes a core of men with long 
experience in intelligence, other men who have worked 
in the foreign or armed services, and a younger group who 
will move on to other positions after a few years on the Board. 
We suggest that it might be helpful when there are vacancies 
to add two or three rotating members who would serve for 
only a year or 80, These men could be borrowed from other 
branches of the government or from universities, If 
properly selected, they could contribute new ideas and 
encourage re-examination of accepted formulae, 

3) Make sure that diverse opinions of other members 
of the community are fully pepkadentea in contributions and 
coordination meetings. Here our knowledge is limited and 


our recommendations are only tentative, We believe that 
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that the problem of overloading is even more serious 
outside CIA than inside. Some representatives apparently 
have little time to inform themselves or to think about the 
problems which they are supposed to discuss. We are 

also uneasy about the tendency to concentrate military 
contributions in the DIA, There are often sharp differences 
of opinion among the armed services. Expression of these 
differences can be very helpful in forming sound judgments, 

4) Call in a small group of consultants when diffi- 
cult problems are to be discussed. We advance this idea 
with some hesitation and we realize that it might cause 
more trouble than it was worth, But it might be a way of 
getting new points of view and of avoiding blind spots. Two 
or three consultants would probably have a greater impact 
than a single outsider who might hesitate to contradict 
the insiders. 

5) Indicate in some way when earlier estimates 
are being quoted or paraphrased. There are many ways 
of doing this (footnotes, quotation marks, etc.) and any 
one would be satisfactory, But attention should be called 
to repetitions, in the hope that this would lead to 
re-examination of accepted positions, 
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of U. S. policy and attitudes on the rest of the world, This 
is a difficult and dangerous task because it can lead to implied 
criticisms of U. S. policy. But it seems to us that failure 
to do this is one reason why preconceived ideas (which are 
still correct) sometimes cause faulty estimates, To be 
specific: if the Kremlin in the early months of 1962 had 
reason to think that the posture and resolution of the U. S. 
were weak, then it should obviously have estimated that the 
risks of the Cuban adventure were not unduly high, Thus 
the accepted doctrine that the USSR would avoid grave risks 
was probably correct; the error was caused by the failure 
to estimate that the Kremlin estimate of U. S. policy had 
changed, 

7) Make more information about attitudes and 
methods of foreign intelligence organizations available 
to estimators, We believe, for example, that the Kremlin 
attaches more importance to certain kinds of evidence, and 
less to others, than the U. S. intelligence community would 
do. We suspect that what information has been acquired 
about Soviet intelligence procedures is not very widely 


disseminated, But if the Kremlin does use a peculiar scale 
SRA 
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in weighing evidence and if this is not generally known, 

then the result is bound to be a faulty estimate, One 

reason for inadequate estimates on Cuba in 1962 was 

that the intelligence community, as.a whole, believed 

that Soviet leaders must realize how dangerous the intro- 
duction of IRBM's and MRBM's would be. Better knowledge 
of Soviet intelligence procedures might have made it possi- 
ble to suggest that the Kremlin was undervaluing the evidence 


which should have led it to this conclusion, 


8) Keep a constant watch for signs that a mutation 


in an existing pattern is about to take place. This is pro- 
bably a superfluous and useless recommendation, Certainly 
everyone is on the look-out for such signs and it is much 
easier to identify them in retrospect than at the moment 
when they occur, Existing patterns can accommodate them- 
selves to a large number of new ingredients and gradual 
changes bring about only gradual shifts in policy. But 

there are some new ingredients so large or so sharp that 
they may break the old pattern and it is not entirely 
impossible to identify these, Again to be specific, in 

1962 the unprecedented situation in Cuba and the Sino -Soviet 


dispute were probably such ingredients. 
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9) Make a few validity studies in depth, At present, 
validity studies usually involve only a comparison between 
anew estimate and the immediately preceding estimate on 
the same topic. Often there has not been enough change in 
the interval between the two papers to reveal errors or biases. 
Our experience in studying the Cuban series of estimates 
leads us to believe that there would be some value in making, 
from time to time, a validity study of a whole group of related 
papers. This might expose persistent biases, rigidities, and 
inconsistencies. We know that one such study of a group of 
Soviet estimates was made some years ago. We suggest 
that more exercises of this sort would be useful, but with the 
caveat that if they became too frequent (say more than one 
or two a year) their value would sharply decrease, We also 
suggest that some of these studies be made by consultants, 
both to reduce the burden on the ONE Staff and to ensure a 


fresh point of view. 


Next to the problem of full discussion comes the problem of 
communication, It seems to be fairly well agreed that estimates often 
teave less impact than they should, and that important ideas are missed 


by hasty readers. Nothing can be done to change the fact that everyone 
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in government has too much to read and that hasty readers will always 
miss significant points. No one style or form of presentation will be 
equally effective with all readers. But perhaps something can be done 
to make it easier for most consumers to read estimates with more 
understanding, 
1) Put key sentences at the beginning of each 

section of the estimate, e.g. ''We believe that the food 

problem in Cuba is (is not) causing unrest,'' Reasons 

for the statement should follow. At present many impor- 

tant statements are made only after a long list of pro's 

and con's and thus lose their full impact. Conclusions, 

as now written, do not always remedy this fault, Perhaps 

a string of key sentences at the beginning of a estimate 

would be better, 

2) Call attention to possible courses of action 

which might have serious consequences, even if the estimate 

is that they are highly unlikely. Too many readers think 

that ''probably not'' equals ''never,"' or that "the possibility 

cannot be excluded" actually means that it is excluded, 

Positive rather than negative formulae would help, but 

we must confess that we are not very good at devising them. 

Perhaps a phrase such as "'there is at least a twenty per 
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cent chance that" approaches what we need. And perhaps 

a little more argumentation showing the advantages which 
might result from the adoption of unlikely courses of action 
would make the consumer take them more seriously. 

3) Emphasize new ideas or facts, Well informed 
readers find much that is familiar in estimates, especially 
in the larger estimates, This lulls them into missing 
useful new material. * Perhaps some typographical device 
(e.g. side-lining) could be used to indicate places where 
significant new ideas and facts are introduced. Or perhaps 
more of the historical and purely descriptive material could 
be put in annexes, 

4) Make sure that the consumer is reminded of 
key items in other estimates. This is probably an 
*For example, there was a very important change in the Cuban 

estimate of 19 September, 1962 (SNIE 85-3-62), Up to that time, all 
estimates and memoranda had asserted that the USSR had little to 
gain by placing MRBM's or IRBM's in Cuba, SNIE 85-3-62, on the 
contrary, pointed out that the Kremlin might see decided advantages 
in such a deployment. But no emphasis was put on the fact that this 
was a changed estimate and we suspect that few readers noticed the 
shift. One of our group, reading the Cuban estimates seriatim 


(which few consumers ever have time to do) still failed to observe 
the fact that there had been a change on this highly significant point. 
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unnecessary caution, but there is some danger that drafters 
of an estimate will assume that consumers remember related 
estimates as wellas they do. For example, in SNIE 85-3-62 
a little more emphasis might have been placed on the fact 
that the estimate of the number of operational Soviet ICBM's 


had recently been sharply reduced. 
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ANNEX A 
'Preconceived Notions" 


In response to one of the requests put before us, we have 


oc 


examined the estimates on Cuba from January 8 to September 19, 1963. 
It is generally acknowledged that there was a failure of intelligence, in 
the sense that USIB did not warn the government that the Soviets might 
be planning to place offensive ballistic missiles in Cuba, and indeed 
suggested that such a decision was unlikely, though not impossible. 
This failure has been miaely attributed to the presence of "preconceived 
notions" which prevented ONE, and in fact the entire intelligence community 
from making a proper judgment, We therefore find it useful to make some 
observations on the function, utility and drawbacks of such preconceived 
notions, Todo so is in fact indispensable to any search for remedies 
designed to minimize these drawbacks, 

In dealing with a major country, intelligence officers appreach 
their task with a set of expectations of likely patterns of behavior. 
This is especially true in the case of the Soviet Union, the most 
frequent target of intelligence, This set of expectations, sometimes 
called preconceived notions, is based on professional knowledge of 
political structures, processes and personalities in the country concerned. 
It is the distillate of years of experience and has been tested by, and 
refined through, repeated intelligencesstimates, It must be understood 
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that the formulation of such a set of expectations about a country's 
likely patterns of behavior is both inevitable and indispensable, It is 
inevitable that, in doing their job, professionals will, more or less 
deliberately, build up the set, It is indispensable because the set is 
a most valuable tool in producing timely, coherent, articulate and, on 
a probability basis, accurate intelligence. If no pattern can be estab - 
lished, then prediction is almost impossible, and significant variations 
can be demonstrated only by comparison with an already accepted norm. 

Nevertheless, these sets of expectations have their limitations 
and drawbacks, The history of intelligence is full of instances in 
which an opponent achieved surprise by adopting a course of action 
that seemed unlikely in view of the set of expectations dominant among 
intelligenge officers, The Japanese attack in Pearl Harbor is one of 
the most conspicious examples. We are not concerned here with what 
may be called "technical surprise," A "technical surprise" is not 
incompatible with the prevalent set of expectations, The surprise occurs 
because the opponent was successful in concealing a particular capability 
or in keeping a particular course of action secret. We are concerned 
with an opponent's behavior that surprises because it is incompatible, 
or seems to be incompatible, with our prevalent set of eee en 
There are three possible causes of this type of surprise. 

First, the opponent's basic pattern of behavior may change as 


a result of changes in leadership or various other important conditions, 
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and our set of expectations may not register this change quickly and 
correctly. Our set, in other words, is out-of-date, Intelligence 
professionals must obviously be on the alert for such changes and, 
generally, take care that their working set of expectations does not 
become obsolete, stale or unduly rigid. 

Second, the opponent may act irrationally, Irrational behavior 
is, of course, very difficult, if not impossible to predict. It may take 
all sorts of directions that make no sense to the rational mind, Only 
if a country's leadership has a record of frequent lapses into irration- 
ality will this observed fact become incorporated into our set of 
expectations. But even then the estimate of future behavior will be 
extremely difficult. 

Third, the opponent may adopt a course of action that seems to 
us to be in conflict with our set of expectations, but actually is not, 

A set of expectations is based on the attitudes and predispositions that, 

we have learned, strongly affect the opponent's behavior, The important 
point to note is that attitudes, though they shape behavior, do not determine 
it. Behavior also depends upon the information which the opponent possesses, 
It is this information which brings his attitudes into play and which gives 

him a basis for calculating the advantages and disadvantages of a course 

of action, Hence, we may go wrong in an estimate, not because our set 

of expectations is faulty, but because we assume that the opponent acts 


on approximately the same information that we have, Clearly, to minimize 
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this source of possible intelligence failure, we require not only a good 
set of expectations but also some idea of the information that the opponent 
uses in his cost-gains calculus, 

We do not think that the failure of the intelligence community 
to predict the Soviet deployment of medium range missiles resulted 
essentially from its set of preconceived notions" about the Soviet Union, 
In our opinion, the community's set of expectations was, at that time at 
least, sophisticated and realistic. According to the set, the Soviet Union 
was considered unlikely to depart from its cautious behavior in the military 
field or to undertake actions involving a high risk of war with the United 
States, That this expectation about Soviet behavior was basically correct 
was, after all, confirmed by the great caution with which Kruchshev acted 
when the missile crisis broke in October, 

Nor do we believe, as some experts have suggested, that the 
Soviet decision was largely irrational, It is true that the Berlin state- 
ment, Chinese criticism, and possible differences of opinion within the 
soviet leadership had put Khrushchev under some pressure to score a 
success in foreign policy, It is also true that during the last year he has 
seemed somewhat less sure of himself than before and has abruptly 
reversed himself on several issues. But while he and other members of 
the top leadership may have been perplexed by current difficulties there 
is no evidence to suggest that they became reckless gamblers, At the 


most, they may have been a little more willing to take a chance than 
Ad 
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they would have been under other circumstances. And a strong case can 
be made for arguing that Soviet behavior was entirely compatible with 
rational decision-making. 

We believe that more important factors in the Soviet decision 
were inadequate information and, resulting from this, an erroneous 
assessment of the situation, They did not have the fund of information 
about the mood and temper of the US government and people which the 
intelligence community thought they must possess, They saw the balance 
of advantages and disadvantages differently from the way Washington 
officials thought they should see it, It is in this area that the intelligence 
community failed, although we defer for the moment the question of the 
extent to which the community can be held accountable for the failure. 

It is fairly clear in retrospect that the intelligence community 
both underestimated the gains that Moscow believed it could derive from 
introducing offensive missiles in Cuba and overestimated the risks which 
the Kremlin thought it ran in adopting this course of action. In reading 
the various estimates seriatim, we were struck by the fact that, although 
the question of the deployment in Cuba of Soviet medium-range missiles 
was raised in a Memorandum to the Director of January 8, 1962, it was 
concluded repeatedly until early September that the Soviet Union could 
not thereby add substantially to its strategic capabilities. The reason 
given was that targets in the United States capable of being reached by 


medium-range missiles from Cuba were already covered by ICBM's 
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deployed on Soviet territory. Only on September 6 were doubts raised 
about this conclusion, The special NIE of September 19, conceded that 
Soviet planners might see some advantage in deploying IRBM's and MRBM's 
in Cuba in order to supplement the limited number of ICBM's believed to 
be operational in the Soviet Union, The delay in coming to this conclusion 
is surprising in view of the fact that it had been estimated several months 
earlier that the USSR had only a small number of operational ICBM's. In 
this situation one obvious way for the Soviet Union to supplement its small 
force of ICBM's was to deploy a proportion of its large supply of medium- 
range missiles within appropriate striking distance of the United States. 
We are not, in fact, sure that such a deployment constituted, objectively, 
a substantial improvement of their strategic position, though Soviet planners, 
of course, may well have thought so. We are sure, however, that the Soviet 
Union would have gained enormously in prestige and, indirectly, greatly in 
the military balance of power and deterrence, if it had succeeded in 
installing offensive’ missiles in Cuba and maintaining them there, Soviet 
leaders, like all rational leaders, must be expected to accept a higher 
level of risk for great gains than for small gains. We note that this was 
not acknowledged in the estimates we examined. 

Even when it was admitted that the Soviet Union might secure 
possible strategic advantages by employing IRBM's and MRBM's in 
Cuba (NIE 85-3-62, September 19, 1962), the conclusion remained that 


the Soviets were unlikely to do so because the risks would be too great 
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and acceptance of so great a risk would represent a sharp departure from 
Soviet practice. This was the crucial view that prevented assigning any 
high degree of probability to the course of action that the USSR pursued. 
However, this view was justified only if the Kremlin assessed the risk 
factor approximately as it was assessed in Washington. The intelligence 
community thought that the risk was very high, It knew it to be so, Yet 
it is fairly clear in retrospect that the Soviet leaders did not share this 
knowledge, that they assessed the risk at a lower level and that, given 
this assessment, their deployment of ''offensive'' missiles in Cuba, did 
not seem unduly risky to them. In that case, their action did not, as 
they saw it, constitute a sharp departure from their normally cautious 
military behavior, 

The extent to which the intelligence community can be held 
accountable for neglecting the possible gap between their information 
and our information is not clear to us, Such gaps are a possible source 
of intelligence failure to which the professional should be alert. In 
retrospect it would certainly have been better if the possibility of this 
gap, and its possible implications, had been acknowledged in the estimates, 
If it had been, it might bave qualified the impact of an erroneous 
conclusion, 

On the other tand, it is patently very difficult for our intelligence 
services to have sufficient knowledge of Soviet intelligence to ascertain 
or predict intelligence failures in the Soviet Union. Moreover, there 
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is reason to believe that much of the intelligence community, including 
ONE, is not provided with such knowledge of Soviet intelligence activities, 
and their organization as is available toa part of the community, Yet 
this is the kind of knowledge which would have been required for a correct 
prediction of the Soviet move in Cuba, 

Moreover, the intelligence officer is also handicapped in speculat- 
ing about national differences in estimating the risk of particular courses 
of action as long as he may not take relevant United States policies fully 
into account, and have adequate information about these policies and their 
implementation to the extent that they are known, or must be assumed to 
be known, to the Soviet Union, 

However, even if these handicaps were removed, or at least 
appreciably diminished, the consumer of intelligence would be wrong 
to expect the intelligence community to be able in all cases to obtain 
sufficient information on the opponent's estimates of the advantages and 
disadvantages of contemplated courses of action. For this reason alone, 
and not necessarily as a result of ''preconceived notions,'' the best 
intelligence service cannot help failing from time to time. It would 
be unfair and utterly unrealistic, in our opinion, to expect otherwise. 

On the other hand, the inevitability of occasional failure does 
not justify either complacency or resignation. No organization stays 
at the same level of performance over a long time -- if only because 
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of changes in personnel -- and there may always be room for improve- 
ment, There are good reasons for reviewing institutional practices 
from time to time with a view of raising the batting average by a few 


per cent, 
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ANNEX B 


Rejected Proposals 


We have considered, and have rejected as impractical, the 
following proposals. 

1) "Devil's advocates", professional dissenters, etc. 
We hope that the intelligence community will always include 
tough-minded, skeptical individuals who will advance unpopular 
ideas and who will query accepted doctrine, But any attempt 
to institutionalize this function would be self-defeating. No 
one can be skeptical all the time and the views of a man who 
was being paid to act as a professional dissenter would not 
carry great weight. 

2) Dual or multiple estimates, For example, one 
group might make an estimate from the Soviet point of view 
while another estimated the same problem from the US point 
of view. The difficulty here is what to do in case of disagree- 
ment, Either some higher group would reach a final decision, 
in which case we would be back where we started, or the 
consumer would make a final estimate, in which case the 
community would have abdicated its responsibility. Moreover, 
there is certain artificiality about this procedure which would 
probably erode its value very quickly, 
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3) Make less effort to reach agreed language in 


estimates, and encourage dissents. It is possible that under 
pressure of time some dissents which might have been helpful 
are never made, But while a few footnotes are useful in alerting 
the consumer to the difficulties of a problem a steady diet of 
footnotes would blunt his attention. If there were too many 
points on which agreement could not be reached the consumer 
eid have to make his own estimate and the community would 
again have abdicated its responsibility, In short, dissents on 
minor points should not be encouraged, and it seems unlikely 
that dissents on major points are, or could be suppressed, 

We note that in the Cuban estimates of 1962 ( and in others 
which we can remember) the worst mistakes were not caused 
by suppressing disagreements or by watering down correct 
judgments in order to obtain an agreed text. They were caused 
by mistaken judgments which were held almost unanimously 
throughout the intelligence community, 

4) Make more effort to keep estimates up-to -date by 
issuing frequent revisions, Carried to an extreme, this would 
obviously lead to a confusion between estimates and current 
intelligence, Even a moderate increase in the number of 
revisions would probably do more harm than good by forcing 
estimators to concentrate on minor fluctuations in policy rather 
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than on longer-range problems, When an estimate has been 


overtaken by events nothing can be gained by issuing a hasty 
revision which simply says that the preceding estimate was 
wrong, When a situation has gone operational revision of 
earlier estimates should wait until there are some firm indica- 
tions of new policies and courses of action, At this point current 
intelligence will give policy-makers the flow of information which 
they require, To refer once more to our case study, we doubt 
that more frequent revisions of the basic estimates on Cuba 
between 1 August and 19 September, 1963 would have changed the 
opinions held by the intelligence community in any important 


way. 
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